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XXXVI. Experiments on a Possible Test of Aesthetic Judgment 

of Pictures 



By Judith Cattell, Josephine Glascock and M. F. Washburn 



The following study represents a preliminary exploration of one 
part of the field mentioned in the title. The materials used were 
thirty-six Cosmos prints in black and white of the following pictures : 

i. Holbein: The Ambassadors. 

2. Thorn : Song Without Words. 

3. Ruysdael : Forest of Oaks. 

4. Delobbe: Noonday Rest. 

5. Holbein : Madonna of the Meyer Family. 

6. Berne-Bellecour : The Intended. 

7. Haquette : Departure of the Fishing Boat. 

8. Hof mann : Jesus and the Woman of Samaria. 

9. Alma Tadema: Sappho. 

10. Naujok: St. Cecilia. 

11. Millet: The Gleaners. 

12. Plockhorst : Christ Blessing Little Children. 

13. Mauve : Sheep, Spring. 

14. Knaus : Holy Family. 

15. Dicksee : Young Handel. 

16. Gardner: David the Shepherd. 

17. Holman Hunt: The Shadow of the Cross. 

18. da Vinci : The Virgin of the Rocks. 

19. Raffaelino : Madonna with Angelic Choir. 

20. Turner : The Fighting Temeraire. 

21. Piglhein: Star of Bethlehem. 

22. Murillo : Children of the Shell. 
2^. Piloty: Thusnelda. 

24. Frise : Reconnaissance by Lions. 

25. Del Sarto : Madonna of the Harpies. 

26. Del Sarto: Entombment. 

27. Murillo : Madonna. 

28. Ver Meer: Young Lady with Pearl Necklace. 

29. Holmes : Kiss Me. 

30. Boughton : Two Farewells. 

31. Douglas : Jersey. 

32. de Hooch : Dutch Interior. 

33. Haquette : The Return to the Port. 

34. Aubert: Cupid's Convention. 

35. Thumann : The Fates. 

36. Gardner: An Improvised Cup. 

These pictures were chosen to represent as wide a range as possible, 
from the sentimentally popular to the technically great. The titles 
and the names of the artists were removed from the pictures before 
they were used. They were then put into the hands of the three 
experts in the Vassar department of art, Messrs. O. S. Tonks, A. E. 
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Bye, and Clarence K. Chatterton. These gentlemen, for whose co- 
operation we wish to express our thanks, were asked to arrange the 
pictures in the order in which they would choose to own the originals 
(the financial value of course being left out of account). From the 
ranks which these three authorities assigned a picture the average 
rank of that picture according to expert judgment was computed; 
and thus a combined ranking, the standard ranking, was obtained for 
the whole series of thirty-six. The order, beginning with the highest, 
of the pictures in this standard ranking was as follows: n and 28 
(equal), 13, 20, 32, 18, 5, 1, 26, 25, 17, 22, 19 and 9 (equal), 3, 27 (here 
a considerable drop in the average rankings occurred, indicating that 
the pictures below this point were considered by the experts as mark- 
edly inferior to those above it), 23, 6, 14, 24, 12, 15, 10 and 7 (equal), 
8, 16, 21, 35, 33, 34, 36, 30, 4, 31, 2, 29. The average deviations of the 
three experts in their judgments of the rank of a particular picture 
ranged from 7.8 places down to .4 of a place, and the average of these 
averages for all the pictures was 3.4 places. 

One hundred and forty-four women student observers were now 
required successively to perform the same experiment; that is, to 
rank the thirty-six pictures in the order in which they would be, 
for their own sake, desirable possessions to the observer making the 
ranking. When each observer had completed her arrangement, she 
was asked the following four questions : " Have you ever studied 
drawing or painting? Have you ever studied the history of art? 
Have you visited many picture galleries? Are you especially inter- 
ested in pictures?" 

The rankings which these one hundred and forty-four observers 
gave the pictures were treated in two ways. First, a rank-difference 
correlation was obtained between each observer's rating of the pictures 
and the expert rating. These correlations varied from +.82 to — .42; 
their average was +.19, with a mean variation of .26. Then, secondly, 
a combined ranking of the pictures for all the one hundred and forty- 
four observers was obtained, by averaging the ranks assigned by 
all the observers to one picture, repeating this process for each of the 
other pictures in turn, and ranking the pictures on the basis of these 
averages. A rank-difference correlation was then found between this 
ranking (which we shall call the 144 O. ranking) and the expert rank- 
ing. The correlation was found to be +.33, with a probable error 
of .10. 

Three small groups were then selected from the total number of 
observers, seventeen in each group. The first group was composed 
of observers who answered "yes" to all the questions, and therefore 
represent those most artistic in training and taste. A combined rank- 
ing from Group I showed a rank-difference correlation with the 
expert ranking of +.49, probable error .09. Group II contained the 
observers who answered " no " to all the questions, and thus might 
be considered the least artistic. The ranking formed from their 
combined judgments gave a correlation with the expert ranking of 
— .11. Group III contained observers who answered "yes" only to 
the last question; that is, they were without artistic training of any 
sort, but were interested in pictures. The correlation of the ranking 
formed from their combined judgments and the expert ranking was 
+.43, nearly as high as that of Group I. 

Some interesting points of difference appeared between these three 
groups as regards the average rank which they assigned to particular 
pictures. We have seen that the experts placed picture 11 (Millet's 
Gleaners) and picture 28 (ver Meer's Young Lady with Pearl Neck- 
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lace) together at the head of the list; picture 13 (Mauve's Sheep) 
stood next; picture 29 (Holmes: Kiss Me) was at the foot, and pic- 
ture 31 (Douglas : Jersey) ranked next but one to the last. Now in 
the combined ranking of all 144 observers, picture 13 stood at the 
head and picture 31 at the bottom. Group I (the artistic), also put 
13 at the top and 31 at the bottom. Group II (the wholly. inartistic), put 
13 at the top; but at the bottom they put number 1, Holbein's Am- 
bassadors, which the experts rated as eighth from the top. Group III, 
the observers who were untrained but interested in pictures, ap- 
proached more nearly to the experts in putting picture 11 at the top; 
31 being at the bottom. 

It thus appears (1) that technical training makes the rankings 
assigned to these thirty-six pictures correlate more closely with the 
rankings assigned them by experts, which might have been expected; 
and (2) that a declared interest in pictures, quite apart from any 
kind of training, has a noticeable effect in producing closer approxi- 
mation of the judgments to those of experts. 

As the results were further studied, it became apparent that the 
rating given by an individual to one or two pictures out of the thirty- 
six was of marked significance as showing that individual's agree- 
ment with the expert standard. This was noticeably true of picture 
28, Ver Meer's Young Lady with Pearl Necklace, which the experts 
put at the top of the list. This picture has but little literary appeal; 
its excellence is in a high degree technical. Its rank in the combined 
144 O. ranking was 20 ; the totally inartistic group II placed it as low 
as 30; the highly artistic group I put it at 12, and the untrained but 
interested observers of group III again showed the effect of their 
natural taste by rating it 'd 16. 

Other indications of the diagnostic significance of an observer's 
reaction to picture 28 were obtained from a study of the indexes of 
correlation between a given individual's ranking of the pictures and 
the expert ranking. There were fifty-nine persons who gave either 
negative correlations or positive correlations less than +.10. The 
rankings which these persons gave to picture number 28 were well 
massed beyond twenty : there were eleven who put it at the bottom 
of the scale and only six who ranked it better than twentieth: the 
average ranking was 28.3. There were forty-one observers who had 
positive correlations of over 40; the ranks which these assigned to 
picture 28 were massed at the upper end, there were only five who 
ranked it beyond eighteen, and the average ranking was 8.8. Thus 
a difference of 19.5 places appeared between the average ranking 
of 28 by the observers whose ranking of the whole series of 36 pic- 
tures failed to correlate or correlated negatively with the expert 
ranking, and the average ranking of 28 by the observers whose ranking 
of the whole series correlated with the expert ranking by more than 
+.40. These results may be compared with those for picture 11, 
Mauve's Sheep, a picture which is attractive in subject as well as in 
technique. The massing of the rankings of this picture was at the 
upper end in the case of both the low correlation groups and the high 
correlation groups ; the average ranking for the low correlation 
group was 6.8, that for the high correlation group 5.3, a difference of 
only 1.5. In other words, everybody liked this picture. 

Further, we looked up the indexes of correlation for the individuals 
who had assigned rankings from 26 to 36, or the last ten ranks, to 
picture 28, and found that out of 53 individuals only five, or 9.4%, had 
correlations of over +.40. Of the 32 individuals who assigned to 
picture 28 a rank somewhere among the first ten places, there were 
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only 12, or 32.4%, who had correlations below +.40. It would seem 
that assigning a low rank to picture 28 is a better indication of poor 
artistic judgment than assigning a high rank to this picture is of 
good artistic judgment. 

Finally, another expression for the diagnostic significance of the 
rank which an observer assigned to picture 28 was sought in the fol- 
lowing way. An average was taken of the indexes of correlation 
obtained between the ranking of the whole series of thirty-six pictures 
and the expert ranking, for all the observers who assigned rank 
number 1 to picture 28. A similar average of the coefficients was 
made for all the observers who assigned rank number 2 to picture 
28, and so on for each of the thirty-six ranks that could be assigned 
to this picture. Of course the number of observers assigning the same 
rank to picture 28 was not equal for all the ranks ; it varied from 
twelve (for the lowest rank, 36) to one (for ranks 16, 19, and 30), 
so these averages were not very significant. But if the judgment 
made on picture 28 were a perfect index of the degree of correlation 
between an observer's ranking of the thirty-six pictures and the 
expert ranking, so that one could predict the correlation from the 
judgment on this one picture, evidently the averages of the indexes 
for observers who made the same ranking of picture 28 ought, when 
arranged in numerical order, to follow the order of the ranks 
assigned to 28. Thus, the average of the correlation indexes of the 
observers who ranked picture 28 as number 1 ought to be the highest 
average; that of the observers who ranked picture 28 as number 2 
ought to be the next highest, and so on down. A rank difference 
index of correlation was accordingly found between the thirty-six 
average indexes and the order 1 to 36, and this coefficient was found, 
in spite of the inadequacy of the averages, to be +.78. Thus it would 
appear that if this series of pictures is used, ranking picture number 
28, ver Meer's Young Lady with Pearl Necklace, anywhere in the 
last ten places means in nine cases out of ten low correlation with 
the expert judgment on all the pictures; ranking it among the first 
ten places means in two cases out of three high correlation with the 
expert judgment on all the pictures. 



